Methods and apparatus for rendering user interfaces and display information on remote client devices

ABSTRACT

A user interface is implemented on a client device remote from a host device. The host device operates an application program that implements a user interface, such as an electronic programming guide or a guide for a personal video recorder, that permits a user to control at least one target device. The host device transfers to the client device an identification of at least one scene. In general, a scene defines an abstract layout for at least one screen display of the user interface. The client device generates at least one screen display for the scene based on its interpretation of the scene. The client device then displays the screen as an implementation of the user interface. Thereafter, a user initiates, using the client device, an operation to control the target device. In response, the target device performs the operation. The host device may also display information at a client device. For example, the host device may transmit information about a media currently playing at the client device.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of U.S. patent application Ser. No.10/391,116, filed Mar. 17, 2003, entitled “Methods and Apparatus ForImplementing A Remote Application Over A Network.”

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention is directed toward the field of network softwareand devices, and more particularly towards rendering user interfaces anddisplays on devices remote from a host device.

2. Art Background

Prior art techniques exist to “remote” applications. In general, aremote application is an application that runs on a first computer butprovides the functionality of the application to a second computer(e.g., implements a user interface) remote from the first computer.Remote application techniques have been used in client—serverenvironments, wherein the application programs are stored on a server,and the client computers accesses the server to obtain functionalityfrom the applications. The X Windows environment remotes applicationssuch that thin client computers, or terminals, access a computer, suchas a server, over a network to obtain the application's functionality atthe terminals. For example, a server may host a word processingapplication. The thin client computer or terminal communicates with aserver to operate the word processing program. The application program,running on the server, implements the user interface at the localcomputer for the underlying application program.

One issue that arises when implementing remote applications is that theremote applications require specific knowledge about the displaycharacteristics of the client computer or terminal. If the client-serverenvironment has many client computers, then the remote application mustknow the requirements of each client computer. This limits the types ofdevices or computers that the remote application can support, orsignificantly increases complexity of the server software to support thevarious types of devices. Therefore, it is desirable to develop softwarethat permits a remote application to operate on a client computer ordevice without requiring the remote application to have any knowledge ofthe client's configuration.

Typically, applications implement a user interface using a userinterface tool kit, sometimes referred to as a widget set, a renderingengine, and underlying hardware to display the user interface. Theapplication provides parameters to the user interface tool kit based onspecifics of the application. For example, some applications definebuttons, toolbars, menus, etc. for use with the application. The userinterface tool kit provides specific layout information for theapplication requirements. For example, the user interface tool kit mayspecify placement of the buttons, toolbars and menus used in theapplication. This layout is sometimes referred to as a logical layout ofthe user interface. The rendering engine, which receives the logicallayout from the user interface tool kit, defines how to translate thelogical layout to a physical representation for rendering on an outputdisplay. For example, if the remote computer display is a graphicsdisplay, then the rendering engine may convert digital data to RGB datafor storage in a frame buffer for rendering on the output display. Theuser interface hardware may include, for a graphics display, a framebuffer, graphics processor and raster scan display for rendering pixeldata.

Typically, to remote an application, the application and user interfacetool kit software are run on the remote computer (i.e., the computerrunning the remote application). The local computer (i.e., the computerproviding the user interface) includes the rendering engine and displayhardware. An interface between the remote computer and local computerdefines specific parameters for displaying information on the localcomputer (e.g., screen resolution, graphics or textual display, colorpalettes supported, etc.). Using this interface, the remote applicationspecifies a logical layout supported by the physical rendering of thelocal client. For example, the remote application may specify, in alogical layout, the toolbar at the top of the output display. In orderto ensure that the toolbar is readable, the remote application knows theoverall resolution of the output display on the local client. Therendering engine at the local client translates the data for thetoolbar, and stores the graphics data in the frame buffer at the localclient. The contents of the frame buffer are thereafter rendered on thelocal client's output display.

Internet technology uses markup languages and Web browsers to displayWeb applications on local computer displays. Using Web browsertechnology, the application running on the web server does not need toknow the specifics of the client display characteristics. However, theapplication logic is typically tuned to a specific display resolution,particularly if the web page contains graphic information. Often timesWeb applications specify to the user at the local computer a viewingresolution for the Web site because the Web application was designed forthat specific resolution. Thus, Web technology still requires theapplication to have specific knowledge of the display characteristics ofa computer that displays Web pages. In addition, user interfaces on theWeb are very disconnected, so as to require splitting the applicationlogic between the server and the client (e.g., Javascript).Alternatively, the Web applications are not very smooth and interactiveapplications. Although Web technology may be useful because most usersview the information from desktop or notebook computers with apre-defined resolution, the technology is not effective for use insystems that integrate devices with different types of displays.Accordingly, it is desirable to develop a remote application technologythat permits using local client displays for remote applicationsregardless of the type of display at the local client. It is alsodesirable to develop applications to utilize remote user interfaces andremote display of information.

SUMMARY OF THE INVENTION

A user interface is implemented on a client device remote from a hostdevice. In one embodiment, the client device comprises a portableelectronic device that includes a graphical display. The host deviceoperates an application program that implements a user interface thatpermits a user to control at least one target device. For example, theuser interface may comprise an electronic programming guide to control atelevision or a guide for a personal video recorder. In otherembodiments, the user interface comprises an interface to control amedia playback device. The host device transfers to the client device anidentification of at least one scene. In one embodiment, the host andclient devices communicate over a wireless network. In general, a scenedefines an abstract layout for at least one screen display of the userinterface. The client device generates at least one screen display forthe scene based on its interpretation of the scene. The client devicethen displays the screen as an implementation of the user interface.Thereafter, a user initiates, using the client device, an operation tocontrol the target device. In response, the target device performs theoperation.

In other embodiments, the host device displays information at a clientdevice. For this embodiment, the client device receives information fromthe host device for display at the client device. The client device mayhave an LCD or a graphical user interface. The host device may transmitinformation about media currently playing at the client device. Forexample, the host device may comprise a media server and the clientdevice may comprise a playback device (e.g., CD player). For thisapplication, the host device transmits information about the musicplaying at the playback device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a media space configured in accordance with oneembodiment of the present invention.

FIG. 2 illustrates one embodiment for integrating devices into a singlemedia space.

FIG. 3 is a block diagram illustrating one embodiment for implementing aremote application.

FIG. 4 is a block diagram illustrating an example abstract scene layout.

FIG. 5 is a block diagram illustrating one embodiment for implementingan abstract scene with widgets.

FIG. 6 illustrates an example screen display generated for a graphicsdisplay.

FIG. 7 illustrates an example screen display generated for a liquidcrystal display (“LCD”).

FIG. 8 is a block diagram illustrating a further embodiment forimplementing a remote application.

FIG. 9 is a block diagram illustrating another embodiment forimplementing a remote application.

FIG. 10 is a block diagram illustrating one embodiment for implementingwidget-based controllers for the remote user interface of the presentinvention.

FIG. 11 is a block diagram illustrating one embodiment for providing amodel for a user interface.

FIG. 12 is a block diagram illustrating another embodiment for a remoteapplication.

FIG. 13 is a flow diagram illustrating one embodiment for a method toremote an application.

FIG. 14 is a flow diagram illustrating one embodiment for implementing auser interface from a remote application.

FIG. 15 illustrates an example user interface for a televisionimplemented on a client device.

FIG. 16 illustrates one embodiment for rendering a user interface of anaudio application on a client device.

FIG. 17 illustrates an application for the remote display of mediainformation from a media server to an A/V receiver.

DETAILED DESCRIPTION Media Convergence Platform:

A media convergence platform provides an efficient and easy way for oneor more users to manage and playback media within a “media space.” Asused herein, a “media space” connotes one or more media storage devicescoupled to one or more media players for use by one or more users. Theintegration of media storage devices and media players into a singlemedia space permits distributed management and control of contentavailable within the media space.

FIG. 1 illustrates a media space configured in accordance with oneembodiment of the present invention. As shown in FIG. 1, the media space100 includes “n” media storage devices 110, where “n” is any integervalue greater than or equal to one. The media storage devices 110 storeany type of media. In one embodiment, the media storage devices 110store digital media, such as digital audio, digital video (e.g., DVD,MPEG, etc.), and digital images. The media space 100 also includes “m”media players 120, where “m” is any integer value greater than or equalto one. In general, the media players 120 are devices suitable forplaying and or viewing various types of media. For example, a mediaplayer may comprise a stereo system for playing music or a televisionfor playing DVDs or viewing digital photos.

As shown in FIG. 1, the media storage devices 110 are coupled to themedia players 120. The media storage devices 110 and the media players120 are shown in FIG. 1 as separate devices to depict the separatefunctions of media storage and media playback; however, the mediaplayers may perform both the storage and playback functions. Forexample, a media player may comprise a DVD player that includes a harddrive for the storage and playback of digital video. In otherembodiments, the storage of media and the playback/viewing of media areperformed by separate devices. For this embodiment, the media players120 playback content stored on the media storage devices 110. Forexample, a video clip stored on media storage device “1” may be playedon any of the applicable “m” media players 120.

The storage devices 110 and media players 120 are controlled bymanagement component 130. In general, management component 130 permitsusers to aggregate, organize, control (e.g., add, delete or modify),browse, and playback media available within the media space 100. Themanagement component 130 may be implemented across multiple devices. Themedia space of FIG. 1 shows a plurality of users 140 to depict that morethan one user may playback/view media through different media players.The system supports playback of different media through multiple mediaplayers (i.e., the system provides multiple streams of mediasimultaneously). The users 140, through management component 130, mayalso organize, control, and browse media available within the mediaspace. The management component 130 provides a distributed means tomanage and control all media within the media space.

FIG. 2 illustrates one embodiment for integrating devices into a singlemedia space. For example, system 200, shown in FIG. 2, may be a homemedia system. For this embodiment, the media space 200 includes at leastone personal video recorder (“PVR”)—media server 210 (i.e., the mediaspace may include many media servers). The media server 210 stores mediafor distribution throughout the media space 200. In addition, the mediaserver 210 stores system software to integrate the components of themedia space, to distribute media through the media space, and to providea user interface for the components of the media space. The PVR-mediaserver 210 may also include one or more television tuners and softwareto record television signals on a storage medium.

As shown in FIG. 2, the PVR-media server 210 is coupled to differenttypes of media players, including one or more televisions (e.g.,television 250) and one or more media players (e.g., audio and videoplayback devices), such as playback device 240. The media playbackdevice may comprise AVR receivers, CD players, digital music players(e.g., MP3), DVD players, VCRs, etc. For this embodiment, the PVR-mediaserver 210 is also coupled to one or more media managers 280 and toexternal content provider(s) 290.

For this embodiment, the PVR-media server 210 executes software toperform a variety of functions within the media space. Thus, in thisconfiguration, the PVR-media server 210 operates as a “thick client.” Auser accesses and controls the functions of the media convergenceplatform through a system user interface. The user interface utilizesthe thick and thin clients, as well as some media players (e.g.,television 250 & media playback device 240). In one embodiment, the userinterface includes a plurality of interactive screens displayed on mediaplayer output devices to permit a user to access the functionality ofthe system. A screen of the user interface includes one or more itemsfor selection by a user. The user navigates through the user interfaceusing a remote control device (e.g., remote control 260). The user,through use of a remote control, controls the display of screens in theuser interface and selects items displayed on the screens. A userinterface permits the user, through use of a remote control, to performa variety of functions pertaining to the media available in the mediaspace.

The components of the media convergence platform are integrated througha network. For example, in the embodiment of FIG. 2, the devices (e.g.,PVR-media server 210, television 250, remote control 260, media player240 and media manager 280) are integrated through network 225. Network225 may comprise any type of network, including wireless networks. Forexample, network 225 may comprise networks implemented in accordancewith standards, such as Ethernet 10/100 on Category 5, HPNA, Home Plug,IEEE 802.11x, IEEE 1394, and USB 1.1/2.0.

For the embodiment of FIG. 2, one or more thin video clients may beintegrated into the media space. For example, a thin video client may becoupled to PVR-media server 210 to provide playback of digital media ontelevision 250. A thin video client does not store media. Instead, athin video client receives media from PVR-media server 210, andprocesses the media for display or playback on a standard television.For example, PVR-media server 210 transmits a digital movie over network225, and the thin video client processes the digital movie for displayon television 250. In one embodiment, the thin video client processesthe digital movie “on the fly” to provide NTSC or PAL formatted videofor playback on a standard television. A thin video client may beintegrated into a television.

The media convergence platform system also optionally integrates one ormore thin audio clients into the media space. For example, a thin audioclient may receive digital music (e.g., MP3 format) from PVR-mediaserver 210 over network 225, and may process the digital music forplayback on a standard audio system. In one embodiment, the thin audioclient includes a small display (e.g., liquid crystal display “LCD”) andbuttons for use as a user interface. The PVR-media server 210 transmitsitems and identifiers for the items for display on the thin audioclient. For example, the thin audio client may display lists of tracksfor playback on an audio system. The user selects items displayed on thescreen using the buttons to command the system. For example, the thinaudio client screen may display a list of albums available in the mediaspace, and the user, through use of the buttons, may command the userinterface to display a list of tracks for a selected album. Then, theuser may select a track displayed on the screen for playback on theaudio system.

The media manager 280 is an optional component for the media convergenceplatform system. In general, the media manager 280 permits the user toorganize, download, and edit media in the personal computer “PC”environment. The media manager 280 may store media for integration intothe media space (i.e., store media for use by other components in themedia space). In one embodiment, the media manager 280 permits the userto perform system functions on a PC that are less suitable forimplementation on a television based user interface.

The media space may be extended to access media stored external to thosecomponents located in the same general physical proximity (e.g., ahouse). In one embodiment, the media convergence platform systemintegrates content from external sources into the media space. Forexample, as shown in FIG. 2, the PVR-media server 210 may access contentexternal to the local network 225. The external content may include anytype of media, such as digital music and video. The media convergenceplatform system may be coupled to external content 290 through abroadband connection (i.e., high bandwidth communications link) topermit downloading of media rich content. The external content may bedelivered to the media convergence platform system through use of theInternet, or the external content may be delivered through use ofprivate distribution networks. In other embodiments, the externalcontent may be broadcasted. For example, the media server 210 may accessexternal content 290 through a data casting service (i.e., datamodulated and broadcast using RF, microwave, or satellite technology).

Remote Applications:

As used herein, a “remote application” connotes software, operating on adevice other than a local device, used to provide functionality on alocal device. As described herein, the techniques of the presentinvention do not require the remote application to possess pre-existinginformation about the characteristics of the local display device (e.g.,display resolution, graphics capabilities, etc.).

In one embodiment, the software system separates the user interface(“UI”) application logic from the UI rendering. In one implementation,the system defines user interface displays in terms of “abstractscenes.” In general, an abstract scene is a layout for a screen display,and it consists of logical entities or elements. For example, anabstract scene may define, for a particular display, a title at the topof the display, a message at the bottom the display, and a list ofelements in the middle of the display. The scene itself does not definethe particular data for the title, message and list. In oneimplementation, the software comprises pre-defined scenes, UIapplication logic, a scene manager, and UI rendering engine. In general,pre-defined scenes describe an abstract layout in terms of logicalentities for a UI display. Typically, the application logic determinesthe scene and provides data to populate the scene based on the logicalflow of the application. For example, a user may select a first itemdisplayed on the current UI display. In response, the application logicselects, if applicable, a new abstract scene and data to populate thenew scene based on the user selection.

The application logic is implemented independent of the scene and the UIrendering. The application logic selects a scene descriptor, to definean abstract layout, in terms of the abstract elements. The applicationlogic then populates the logical elements with data, and transfers theabstract layout (scene descriptors) with data to the display client. Ascene manager, running on the local client, interprets the scenedescriptors based on the display capabilities of the display client. Forexample, if the display for a display client is only capable ofdisplaying lists, then the scene manager translates the scene with datato display only lists. This translation may result in deleting someinformation from the scene to render the display. The scene manager mayconvert other logical elements to a list for display on the LCD display.The UI rendering engine renders display data for the scene with displayelements particular to the output display for the display client. Thedisplay elements include display resolution, font size for textualdisplay, the ability to display graphics, etc. For example, if theoutput device is a television screen, then the UI rendering enginegenerates graphics data (i.e., RGB data) suitable for display of thescene on the television screen (e.g., proper resolution, font size,etc.). If the output display is a liquid crystal display (“LCD”), the UIrendering engine translates the scene logical entities to a formatsuitable for display on the LCD display.

A user interface implementation that separates the UI application logicfrom the UI rendering has several advantages. First, the applicationlogic does not require any information regarding the capabilities of theoutput display. Instead, the application logic only views the UI displayin terms of logical entities, and populates data for those logicentities based on user input and logical flow of the user interface.Second, this separation permits a graphical designer of a user interfacesystem to easily change the scenes of the user interface. For example,if a graphical designer desires to change a scene in the user interface,the graphical designer only changes the mapping from abstract tophysical layout of the scene. During runtime, the application logicreceives the revised scene descriptor, populates the revised scenedescriptor with data via slots, and transmits the scene descriptor withdata to the local client. Software on the local client determines thosedisplay elements to display the scene based on the device's display.Thus, a change to the scene does not require a change to the displayelements particular to each output display because the conversion fromthe scene to the display elements occurs locally.

In one embodiment, the media convergence platform permits implementinguser interface software remote from a device. In one implementation, theapplication logic is executed on a device remote from the devicedisplaying a user interface. The device displaying the user interfacecontains the UI rendering software. For this implementation, the dataand scenes for a user interface (e.g., scene descriptors) exist on aremote device. Using this implementation, the scene interface (interfacebetween the scene descriptors and the application logic) is remote fromthe device rendering the display. The remote device (e.g., server) doesnot transfer large bitmaps across the network because only scenedescriptor information with data is transferred. This delineation offunctions provides a logical boundary between devices on a network thatmaximizes throughput over the network. In addition, a remote devicehosting the application logic does not require information regardingdisplay capabilities of each device on the home network. Thus, thisimplementation pushes the UI rendering software to the device renderingthe images, while permitting the application logic to reside on otherdevices. This architecture permits implementing a thin client in a mediaconvergence platform because the thin client need not run theapplication logic software. In addition, the architecture permitsimplementing a “thin application server” because the application serverdoes not need to know about every possible rendering client type.

FIG. 3 is a block diagram illustrating one embodiment for implementing aremote application. For this example embodiment, a remote application310 includes scene descriptors 320 and application logic 330. Forexample, remote application 310 may comprise a media server withconsiderable processing capabilities, such as a computer or set-top box.A client device, 370, has a display 360, for displaying information to auser (e.g., displaying data to implement a user interface), a renderingengine 355, and a scene manager 350. The rendering engine 355 receives,as input, data model from scene manager 350, and generates, as output,display data. The display data is a type of data necessary to render animage on the display 360. For example, if the display 360 comprises agraphics display, then display data includes information (e.g., RGBdata) to render a graphical image on a display.

FIG. 3 illustrates separating a UI rendering, implemented on a clientdevice, from application logic implemented on a remote device (310). Inan example operation, a list of objects (e.g., musical albums) may bedisplayed on display 360. In this example, the user may select an albumfor playback. A scene descriptor (320) may define an abstract layout forthis application. For example, the scene descriptor may define a list ofaudio track elements and control information. The application logic 330receives the scene descriptor. The application logic 330 populates theelements of the scene descriptor with data particular to the selection.Thus, for this example, application logic 330 populates the list ofaudio track elements with the names of the audio tracks for the albumselected by the user. The application logic 330 then transmits, throughinterface 340, the scene data to the scene manager 350 on client 370.The scene manager 350 converts the scene elements with data to thedisplay elements. The rendering engine 355 generates data in a formatsuitable for display on display 360. For example, if display 360 is anLCD display, then rendering engine 355 generates a textual list of audiotracks. In another example, if display 360 is a graphics display, thenrendering engine 355 generates graphics data (e.g., RGB), for the listof audio tracks.

In one embodiment, the techniques use “abstract scenes”, defined byscene descriptors, to implement a user interface. In one embodiment,each application communicates in terms of at least one scene descriptor.A scene descriptor, in its simplest form, may constitute a list (e.g., alist scene). In general, a scene descriptor defines a plurality of slotsand the relative locations of those slots for rendering the scene on anoutput display. The slots of a scene provide the framework for anapplication to render specific information on a display. However, anabstract scene defined by a scene descriptor does not define specificcontent for a slot. The abstract scene is developed in the applicationlayout section on the remote computer (i.e., the computer operating theremote application).

In one embodiment, the system divides labor between the remoteapplication computer and the local display computer through use of scenedescriptors. Specifically, the remote application communicates the scenedescriptor, in terms of logical coordinates, to the local displaycomputer. The local display computer translates the scene descriptorbased on its underlying display capabilities. In other embodiments, theremote application may define additional information about a scene, soas to shift more UI operations to the remote application. In yet otherembodiments, the remote application may provide less information about ascene, thereby assigning more UI operations to the local clientcomputer.

As an example, a scene descriptor may include one or more titles, amessage, and a list of elements. FIG. 4 is a block diagram illustratingan example scene descriptor. As shown in FIG. 4, the example sceneincludes a plurality of slots (i.e., A, B and C). A slot_(A), located onthe top of the scene, may be used to display a major title (e.g., thetitle of the application). Slot_(B), located in the center of the scene,includes a plurality of elements, 1-n, to display a list. For example,slot_(B) may be used by the application to display menu items. For thisexample, each subcomponent (e.g., slot_(B1), Slot_(B2) . . . slot_(Bn))may represent a menu item. In one application, the menu items comprisemedia items (e.g., music, video, etc.) available in a media system. Thenumber of menu items displayed may be variable and dependent upon thedisplay capabilities of the local computer. The third slot shown in FIG.4, slot_(C), is displayed in the lower left corner. The remoteapplication may use slot_(C) to display an icon (e.g., a logo for theremote application software publisher).

In one embodiment, the remote application constructs a list of elementsfor a scene descriptor, which includes data for display in the slots,and transfers the list of elements in a block defined by the interface(e.g., interface 340, FIG. 3) to the local display device. In oneembodiment, the remote application interrogates the scene (at theclient) to determine the number of visible elements for display, andthen retrieves the list items for those visible elements. For example,the list elements may include a data model, abstract interface model,raw string, etc.

In one embodiment, “widgets”, a software implementation, are used in theuser interface For this embodiment, an abstract scene is implementedwith a collection of widgets. A widget corresponds to one or more slotson an abstract scene. In one implementation, a widget comprises acontroller, model, and view subcomponents. A view is an interpretationof the abstract scene suitable for a specific display. For example, afirst view of an abstract scene may be suitable for rendering on agraphical display, and a second view of an abstract scene may besuitable for rendering the abstract scene on an LCD display. The modelprovides the underlining data for slots of an abstract scene. Forexample, if a slot consists of a list of menu items, then the model forthat slot may include a list of text strings to display the menu items.Finally, a controller provides the logic to interpret user interfaceevents (i.e., user input to the user interface). For example, if a userselects a menu item displayed on the user interface, an event isgenerated to indicate the selection of the item. The controller providesthe logic to interpret the event, and initiate, if necessary, a newmodel and view.

FIG. 5 is a block diagram illustrating one embodiment for implementingan abstract scene with widgets. The example abstract scene of FIG. 4 isshown in FIG. 5. A widget corresponds to each slot on the abstractscene. Specifically, widget_(A) is instantiated for slot_(A), widget_(B)is instantiated for Slot_(B), and widget_(C) is instantiated forslot_(C). Also, as shown in FIG. 5, each widget (A, B and C) includes acontroller, model and view. Note that slot_(B) on the abstract interfaceincludes a number of subcomponents. Widget_(B) may be configured torender slot_(B), and its subcomponents.

FIGS. 6 and 7 illustrate two different example screens supported by thetechniques of the present invention. An example user interface display(e.g., screen) for a graphics display is shown in FIG. 6. A secondexample screen for a liquid crystal display (“LCD”) is shown in FIG. 7.The example screens utilize the example scene descriptor of FIG. 4. Thetext string “Home Media Applications” is populated in Slot_(A) (FIG. 4)on screen 600 of FIG. 6 and on screen 700 of FIG. 7. However, theunderlying widget for screen 600 presents the text string, “Home MediaApplications”, in a box. For the LCD display 700, the text string “HomeMedia Applications” is displayed on the first line of the display.Slot_(B) (FIG. 4) contains a plurality of elements. For this example,the elements represent menu items (i.e., home media applicationsavailable). Each element (i.e., “Music Jukebox”, “Photo Albums”, “VideoClips”, and “Internet Content”) is individually displayed in a graphicsbox on display 600. For display 700, the menu items are displayed onindividual lines of the LCD display. A third slot, Slot_(C), for thescene descriptor (FIG. 4) is displayed on screen 600 as a graphicalsymbol. The LCD display 700 (FIG. 7) can't display graphics, andtherefore the graphics symbol is not displayed. The example userinterface displays of FIGS. 6 and 7 illustrate two different screensgenerated for the same remote application.

FIG. 9 is a block diagram illustrating another embodiment forimplementing a remote application. The application logic 910 implementsthe functionality of an application program, and display client 940implements a user interface. As shown in FIG. 9, to implement theapplication, application logic 910 communicates with display client 940in a manner to divide functionality between application logic 910 anddisplay client 940. For this embodiment, display client 940 performsmore functions than a purely “thin” client (i.e., display client 940 maybe described as a “thicker” client). The display client 940 includesmodules to implement a user interface for the application specific tothe display capabilities of the display client. To this end, displayclient 940 includes scene manager 945, scene 950, slots 955, andpre-defined scenes 365. A widget, 960, includes model 930, view 965, andcontroller 970 components implemented in both application logic 910 anddisplay client 940. The dashed line around widget 960 indicates that thewidget is implemented across both application logic 910 and displayclient 940. Specifically, display client 940 implements controller 970and view 965 portions of widget 960. The model portion of widget 960 isimplemented on application logic 910 (i.e., model 930). As shown in FIG.9, application logic 910 also includes scene descriptors 920.

In operation, application logic 910 selects an abstract scene for theuser interface. To this end, application logic 910 interrogates displayclient 940 to determine the scenes supported by display client 940(i.e., scenes available in pre-defined scenes 365). The applicationlogic 910 transmits a scene descriptor (one of scene descriptors 920) todisplay client 945 to identify the abstract scene. Based on the scenedescriptor, the scene manager module 945 instantiates a scene for theuser interface. The instantiated scene is depicted in FIG. 9 as scene950. The scene 950 aggregates through the slots 955 to compose a userinterface screen. The slots 955 of scene 950 are populated through useof widget 960. Specifically, input events, input from the user throughthe user interface, are processed by controller 970. The model tosupport the slots is provided from the model 930 in application logic910. Finally, the view of each slot is supported by view module 965,implemented by the display client 940.

FIG. 8 is a block diagram illustrating a further embodiment forimplementing a remote application. For this embodiment, widget 860,supporting the slots for a scene, is implemented entirely on applicationlogic 810. Thus, for this embodiment, display client 840 may becharacterized as a “thin” client. Similar to the embodiment of FIG. 9,application logic 810 interrogates display client 840 to determine theavailable scenes (i.e., scenes available in predefined scenes 865). Toimplement the user interface on display client 840, application logic810 transmits, over a network, a scene descriptor to identify anabstract scene. Based on the scene descriptor, scene manager 845instantiates a scene (e.g., scene 850). The slots for the scene arepopulated through use of widget 860. Specifically, input events,received from the user interface, are propagated, across the network, tocontroller module 870. Model 830 provides data to support the userinterface slots, and view module 865 supports the view module 865. Forthis embodiment, both the model and the view modules are implemented onapplication logic 810. As shown in FIG. 8, the view is communicated backto display client 850. The scene 850 aggregates through the slots 855 togenerate a screen for the user interface. The software for thecontroller portion of a widget may reside locally on a client, or may beinvoked across the network from a remote network device.

FIG. 10 is a block diagram illustrating one embodiment for implementingwidget-based controllers for the remote user interface of the presentinvention. For this example, local display device 1030 instantiates awidget, widget_(A), to control and render one or more slots of theabstract scene. For this example, widget_(A) consists of softwarelocated on both the local display device 1030 (local controller 1040)and on the client network device 1010 (remote controller 1025). For thisimplementation, the local controller 1040 processes certain events forwidget_(A). Typically, local controller 1040 may process simple eventsthat are less driven by the application logic. For example, an event maybe generated when a user moves the cursor from one item on a list toanother. In response to this action, the user interface may highlighteach item to indicate the placement of the user's cursor. For thisexample, a widget may use local controller 1040 to process the event toinitiate a new model and view (e.g., render a highlighted menu item onthe list).

Other events may require more sophisticated operations from theunderlining remote application. In one embodiment, to accomplish this,the remote application (1020), operating on client network device(1010), instantiates a remote controller (1025). In other embodiments,remote controller 1025 may not be a separate object, but may be part ofprocedural code within remote application 1020. As shown in FIG. 10,widget_(A), operating on local display device 1030, propagates an eventto remote controller 1025 through interface 1050. In one embodiment,widget_(A) uses a remote procedure call (“RPC”) mechanism to invokeremote controller 1025 on remote network device 1010 for operation atthe local display device 1030. For example, widget_(A) may receive anevent from a user to select an application displayed on the screen froma menu list of available applications. In response to the event, theremote controller 1025 may generate a top menu screen for the newapplication. The new top menu screen may require a new scene descriptor,or may use the existing scene descriptor.

Data may be supplied to a local display device either locally or fromacross the network. FIG. 11 is a block diagram illustrating oneembodiment for providing a model for a user interface. For thisembodiment, local display device 1130 operates remote application 1120operating on remote network device 1110. The remote application 1120instantiates a data model object 1125. For this example, data modelobject 1125 provides an interface to a data store 1140. The data store1140 may reside on the remote network device 1110, or it may resideanywhere accessible by the network (e.g., another network device or aservice integrating the data store from an external source to thenetwork). For this embodiment, the controller (not shown) interprets anevent, and invokes the data model object in accordance with theinterpretation of the event. For example, in a media system, thecontroller may interpret an event that requests all available musicaltracks within a specified genre. For this request, data model object1125 may generate a query to database 1140 for all musical tracksclassified in the specified genre. As shown in FIG. 11, data modelobject 1125 communicates the model (data) to local display device 1130through interface 1150.

In other implementations, the model may comprise a text string. Forexample, a current UI screen may consist of a list of all high-levelfunctions available to a user. For this example, a user may select afunction, and in response, the system may display a list, which consistsof text strings, of all sub-functions available for the selectedfunction. In another embodiment, the data model may be provided as ahandle to the user interface implementation.

FIG. 12 is a block diagram illustrating another embodiment for a remoteapplication. For this embodiment, the widgets for a local display deviceare highly distributed. For example, one or more widgets may beimplemented across multiple devices (e.g., multiple applicationservers). For the example of FIG. 12, three widgets are utilized toimplement the local display device 1175. Although the example of FIG. 12shows three widgets distributed among three devices, any number ofwidgets implemented over any number of devices may be implementedwithout deviating from the spirit of scope of the invention. A firstwidget, widget, comprises view₁ (1180) implemented on local displaydevice 1175. As shown in FIG. 12, the model—controller (1157)implementation for widget₁ is implemented on application server₁ (1155).Widget₂, which has a view component (1185) implemented on local displaydevice 1175, implements the model—controller component (1162) on aseparate device (i.e., application server 2). A third device,application server 3 (1170), implements the model—controller component(1172) for a third widget. As shown in FIG. 12, the view component(1190) for the third widget is implemented on local display device 1175.

The remote application technology of the present invention supportshighly distributed applications. For the example shown in FIG. 12, asingle display (e.g., local display device 1175) may support threedifferent remote applications. Information for the three applications isintegrated on a single display. Since the view component of the widgetis implemented on the local display device, the view of information forthe remote applications may be tailored to the particular displaydevice.

FIG. 13 is a flow diagram illustrating one embodiment for a method toremote an application. First, the remote application defines a set ofwidgets and selects a scene descriptor that describes an abstract scene(block 1210, FIG. 13). The remote application obtains a handle to thelocal display device (block 1220, FIG. 13). The handle provides a meansfor the remote application to communicate with the local display device.The remote application queries the local display device for a sceneinterface for the defined scene descriptor (block 1230, FIG. 13). Ingeneral, the scene interface provides a means for the remote applicationand local display device to communicate in terms of an abstract scene.To this end, the scene interface defines slots for the abstract scene.For example, to provide a data model, the remote application populates adata structure in the scene interface.

The local display device instantiates software to locally implement theabstract scene and one or more components of one or more widgets (block1240, FIG. 13). As described above, a widget may incorporate all orportions of the controller, model, view subcomponents of a widget. Inone embodiment, the remote application transfers the initial scene data(model) to the local display device through the scene interface (block1250, FIG. 13). In turn, the local display device renders the initialscene using the data model (block 1250, FIG. 13). When the user submitsinput to the user interface (e.g., the user selects a menu item from alist), the widget executes control logic based on the user input (i.e.,event) (block 1270, FIG. 13). In some embodiments, the controller may beimplemented by one or more remote applications. For example, the localdisplay device may include two widgets. The controller of a first widgetmay be implemented on a first application server, and the controller forthe second widget may be implemented on a second application server. Inother embodiments, the local display device may implement thecontroller. A new model is implemented based on interpretation of theuser event (block 1280, FIG. 13). In some embodiments, the model may besupplied by one or more remote applications. The local display devicemay be implemented with four widgets. For this example, a first remoteapplication, running on a first application server, may supply the modelfor two widgets and another remote application server, running on asecond application server, may supply the model for the other twowidgets. Also, the widget renders a new scene with the data modelsupplied (block 1290, FIG. 13).

FIG. 14 is a flow diagram illustrating one embodiment for implementing auser interface from a remote application. The process is initiated whenthe user interface receives input from a user (block 1310, FIG. 14). Awidget, corresponding to the slot on the abstract scene, interprets theuser input and generates an event (block 1320, FIG. 14). For example, ifthe user selects a menu item displayed in a slot on the abstract scene,then the widget that manages the corresponding slot generates an eventto signify selection of the menu item.

If the widget controller for the event is local, then the widgetcontroller, operating on the local display device, interprets the event(blocks 1330 and 1350, FIG. 14). Alternatively, if the widget controlleris implemented remotely, then the widget remotes the event across thenetwork (block 1340, FIG. 14). For example, the controller for thewidget may reside on one more computers remote from the local displaydevice. If the widget interprets the event locally and the event doesnot require an external data model, then the widget supplies the datamodel (blocks 1360 and 1370, FIG. 14). If the event does require anexternal data model from one or more remote devices (over the network)or the widget remotes the controller over the network to one or moreremote devices, then one or more remote applications iterate through theactive widgets on the remote scene and provide a data model to the localdisplay device (blocks 1360 and 1380, FIG. 14). Using the data model,the scene is rendered at the local display device (block 1390, FIG. 14).

Two-Way Universal Remote Controller

The present invention has application to configure a two-way universalremote controller. In general, the remote controller may be configured,on the fly, to control any device on the network. Specifically, a userinterface, operating as a remote application, is implemented on a remotecontroller to control a target device. For this application, the remoteapplication (e.g., user interface) runs on a host computer device, andthe remote controller, operating as a client device, renders the userinterface. As a first “way” of communications, the remote controller,operating as the rendering client, communicates with the remoteapplication to implement the user interface. Then, as a second “way” ofcommunication, the remote controller communicates with the target deviceto control the target device.

The remote controller may comprise a graphical display to implement agraphical user interface. For example, the remote controller orrendering client may comprise a personal digital assistant (“PDA”), atablet personal computer (“PC”), a Java® phone, any portable deviceoperating Windows® CE, or a television user interface. In otherembodiments, the remote controller or rendering client may beimplemented using a character display.

In one embodiment, the two-way remote controller implements a televisionuser interface. For example, the remote application may implement anelectronic programming guide (“EPG”) that operates as a user interface.Through use of the EPG, a user may select programming for viewing on atelevision. FIG. 15 illustrates an example user interface for atelevision implemented on a client device. For this example, television1400 displays an electronic programming guide (“EPG”). The EPG permits auser to select programming for viewing on a television. For thisembodiment of an EPG, multiple channels are displayed in a firstvertical column. For the example EPG displayed in FIG. 15, channels500-506 are shown. The user may view, on the EPG, additional channels byscrolling up or down, through use of a remote control device, the listof channels displayed in the EPG. Additional vertical columns aredisplayed that indicate time slots for the channels (e.g., the timeslots 4:30, 5:00 and 5:30). As shown in FIG. 15, the EPG displays,beneath the time slot columns, the name of the program playing duringthat time slot. For example, “Movie Showcase” is playing on channel 500in time slots 4:30-5:30. On the top of the EPG, information about aselected program is shown. For the example shown in FIG. 15, the program“April Morning” is selected, and information about “April Morning”, suchas type of programming, genre, actors, short description, and programoptions, is displayed.

For the example of FIG. 15, a portion of the EPG, displayed ontelevision 1400, is rendered on a client device 1410. For this example,client device 1410 comprises a graphics display 1420. The client devicedisplay 1420 may also be sensitive to pressure (e.g., touch sensitive)to permit user input through the display screen. For this example,client device display 1420, which renders the user interface, is smallerthan the display of television 1400. Thus, it is not practical oreffective to map the user interface directly to the client devicedisplay 1420. Therefore, in rendering the user interface on the clientdevice display, only essential information is displayed. For the examplerendering shown in FIG. 15, the television column, displaying channels500-506, and one time slot column, with corresponding programminginformation, are displayed. A user may scroll, either vertically orhorizontally, to view different channels or time slots of programming.The example client device rendering of the user interface does notdisplay metadata of a selected program (e.g., the information regarding“April Morning” shown in television 1400). Thus, the television userinterface is rendered on a client device with a small display. Theconcepts of the present invention may also be applied to generate a userinterface for a personal video recorder (“PVR”) or a digital videorecorder (“DVR”).

The present invention also has application for rendering non-graphicaluser interfaces on a client device. For example, a user of home medianetwork 200 (FIG. 2) may desire to use remote control 260 to controlplayback device 240. The playback device 240 may not have a graphicaluser interface. Instead, the playback device 240 may have acharacter-based display. FIG. 16 illustrates one embodiment forrendering a user interface of an audio application on a client device.In one embodiment shown in FIG. 16, the target device may comprise anaudio-video receiver (“AVR”). The AVR displays information on acharacter-based display 1510. For example, display 1510 displays the AVRinput source (e.g., tuner) as well as additional information (e.g., bandand station currently tuned). The user interface on the AVR 1500 furtherincludes buttons 1520 for user input (i.e., the AVR 1500 may also have aremote control specific to the AVR).

For this embodiment, a client device 1530 renders a user interface forthe AVR. The client device is not manufactured specifically for the AVR.Instead, a remote application, residing on AVR 1500 or elsewhere,remotes the user interface of the AVR (i.e., target device) to theclient device 1530. For this embodiment, the client device 1530comprises a graphical display 1540, also used as a user input device(i.e., user touches the screen to interact with the user interface). Forthis AVR example, the client device 1530 renders, as part of the userinterface, tuning control 1550 and volume control 1570 for control ofthe tuning and volume on AVR 1500, respectively. The client device 1530also renders, as part of the user interface, additional information thatspecifies the source of the AVR and the station currently tuned as wellas a volume indicator 1560. The host application and client device mayalso be configured to change the user interface based on the mode ofoperation of the AVR. For purposes of explanation, a single AVRapplication is presented, however, the target device may comprise acompact disc (“CD”) device, a digital video disc (“DVD”) device, adigital music playback device, or any device that provides mediaservices to the network.

The techniques of the present invention have application to render auser interface of a media convergence platform to a client device. Auser interface for a media convergence platform may present differenttypes of media within a single user interface. In one embodiment, theuser interface is television based. For example, the user interface maydisplay, on a television display, selectable items to represent a musicapplication, a photo albums application, and a video application. Theuser selects an item displayed on the television display to invoke anapplication. The music application permits a user to select musicavailable within the media convergence platform, and to playback themusic through a device accessible through the network. The photo albumsapplication permits a user to select one or more photos available withinthe media convergence platform, and to view the photos through a devicein the media convergence platform. The video application permits a userto select one or more videos or video clips available within the mediaconvergence platform and to playback the video/video clips through adevice accessible on the network.

In one embodiment, a two-way remote control device is configured toimplement a user interface for a media convergence platform. Forexample, a host computer device, such as a media server (e.g., PVR-mediaserver 210, FIG. 2), may run an application program to implement themedia convergence platform user interface. In one embodiment, the mediaconvergence platform user interface is television based (e.g.,implemented on television 250, FIG. 2). In another embodiment, the mediaconvergence platform user interface is implemented on a personalcomputer (e.g., implemented on personal computer 250, FIG. 2). Theapplication program may remote the user interface to a client device,such as a PDA. Using the client device, the user may control targetdevices on the network through a rendition of the media convergenceplatform user interface. This allows the user the ability to use theuser interface of the media convergence platform even though the primarydisplay for the user interface is not available or convenient. Forexample, the media convergence platform may primarily use a television,to display screens for the user interface, and a television remotecontrol to accept input from the user. For this example, a user of themedia convergence platform may desire to remote the user interface to aclient device because the television is not available or accessible tothe user (e.g., the television is not in the same room as the user).Using the client device as a two-way remote, the user may proceed tocontrol a target device (e.g., an audio player in the room with theuser).

In one embodiment, the host computer device remotes a user interface toa client device to control another target device on the network. Forexample, in the home network of FIG. 2, the PVR-media server 210 may runa user interface for playback device 240. Under this scenario, thePVR-media server 210 remotes the user interface for playback device 240to remote control 260. As another example, media manager 280 may remotea user interface for displaying photos to remote control 260. In turn,remote control 260 may be used to control an application to view photoson television 250. Accordingly, as illustrated by the above examples, anapplication, which implements a user interface for a remote device, mayreside anywhere on the network to control any other device on thenetwork.

Remote Display Applications:

The present invention has application to render display information at aclient device from an underlying remote application. An applicationprogram, operating on a remote device (e.g., media server), may remoteinformation to a client device (e.g., playback device in a homenetwork). The client device may display the information on a display,such as an LCD. In one embodiment, the client device renders displayinformation to identify media or information about the media. Forexample, the client device may display information about media playingthrough the client playback device.

FIG. 17 illustrates an application for the remote display of mediainformation from a media server to an A/V receiver. For this example, amedia server operates a program to browse, identify and select media.The media server may operate a media convergence platform user interfacefor browsing and selecting, for playback, video, audio and photos. Forthe example shown in FIG. 17, media server 1620, with output device1600, is currently operating an audio application. For this application,output device 1600 displays a list of available audio selections (e.g.,albums, tracks, artists, etc.). In addition, output device 1600 displaysan identification of the audio track currently playing (e.g., NowPlaying Bruce Spingsten). An A/V receiver 1610, coupled to media server1620, is used as a playback device for media server 1620. A/V receiveralso includes a display 1630 (e.g., LCD). For this example, software,operating on media server 1620, remotes display information on A/Vreceiver 1610 for display on display 1630. Specifically, for thisapplication, the A/V receiver displays information about the currentmusic playing.

In another embodiment, the client device displays video information at aclient device. For example, a DVD player may be configured as a playbackdevice to play video from a source on the network (e.g., media server).For example, a media server may supply video (e.g., DVD) for playback atthe DVD player. For this example, a display on the DVD player maydisplay the name of the DVD currently playing as well as additionalinformation about the DVD. In other embodiments, a client device maydisplay information about photos or any other type of media.

Although the present invention has been described in terms of specificexemplary embodiments, it will be appreciated that various modificationsand alterations might be made by those skilled in the art withoutdeparting from the spirit and scope of the invention.

1. A method for implementing a user interface on client device remotefrom a host device, said method comprising the steps of: operating anapplication program on a host computer device, said application programcomprising an underlying user interface to control at least one targetdevice; transmitting, from said host computer device to a client device,an identification of at least one scene, said scene defines an abstractlayout for at least one screen display of said user interface;generating at least one screen display for said scene based on aninterpretation of said scene at said client device; displaying, on saidclient device, said screen display of said user interface; receivinginput, at said client device, from a user to initiate at least oneoperation at said target device; and performing said operation at saidtarget device in response to control from said client device.
 2. Themethod as set forth in claim 1, further comprising coupling said hostcomputer device and said client device through a wireless network. 3.The method as set forth in claim 1, wherein said client device comprisesa portable electronic device that includes a graphical display.
 4. Themethod as set forth in claim 1, wherein said user interface comprises anelectronic programming guide to control a television.
 5. The method asset forth in claim 1, wherein said user interface comprises a guide fora personal video recorder.
 6. The method as set forth in claim 1,wherein: said user interface comprises an interface to control a mediaplayback device; and said target device comprises a media playbackdevice.
 7. The method as set forth in claim 1, wherein said targetdevice comprises said host computer device.
 8. The method as set forthin claim 1, wherein said target device comprises a personal videorecorder (“PVR”) server.
 9. The method as set forth in claim 1, whereinsaid target device comprises a media server.
 10. The method as set forthin claim 1, wherein said target device comprises a television.
 11. Amethod for displaying information on client device remote from a hostdevice, said method comprising the steps of: transmitting, from a hostdevice to a target device, an identification of at least one scene, saidscene defines an abstract layout for at least one screen display;generating at least one screen display for said scene based on aninterpretation of said scene at said target device; receivinginformation at said target device; and displaying, on said targetdevice, said information within said screen display.
 12. The method asset forth in claim 11, further comprising the steps of: accessing mediaat said host device; transferring said media from said host device tosaid target device; and playing said media at said target device. 13.The method as set forth in claim 12, wherein said media comprises video.14. The method as set forth in claim 13, wherein said informationcomprises an electronic programming guide.
 15. The method as set forthin claim 13, wherein said information comprises a guide for a personalvideo recorder.
 16. The method as set forth in claim 12, wherein saidmedia comprises audio.
 17. The method as set forth in claim 16, whereinsaid information comprises information to identify said audio.
 18. Themethod as set forth in claim 11, wherein the step of displaying saidinformation within said screen display comprises the step of displayingsaid information on a liquid crystal display (“LCD”) of said targetdevice.
 19. The method as set forth in claim 11, wherein the step ofdisplaying said information within said screen display comprises thestep of displaying said information on a graphics display of said targetdevice.
 20. The method as set forth in claim 11, further comprisingcoupling said host computer device and said client device through awireless network.